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(54) Lane image processing system for vehicle 

(57) A lane image processing system for a vehicle to 
recognize the lane on which the vehicle travels. The sys- 
tem has a CCD camera mounted on the vehicle for view- 
ing a roadway scene ahead of the vehicle to output im- 
age data of the roadway scene including the lane on 
which the vehicle travels. Obstacles such as preceding 
vehicle are detected and a region for image data 
processing is determined (S1 8) so as to avoid the ppsi- . . 
tion of the obstacle and the lane on which the vehicle 
travels is recognized based on the processed result. 
More specifically, lane boundary positions estimated up 
to a preceding cycle are stored as historical information 
(S14) and in the region Hough transformation is limited 
in areas determined based on the historical information \ 
and the horizon in the roadway scene. 
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Description 

This invention relates to a system for processing 
lane images for a vehicle, more particularly to a system 
for processing lane images for a vehicle enabling a ve- 
hicle to accurately identify vehicle lanes while traveling 
on an expressway or in some other such environment 
including multiple vehicle lanes. 

One existing technology relating to the processing 
of lane images for a vehicle is the method described in 
U.S. Patent No. 4,970, 653, for determining which re- 
gions of images received via a TV camera or the like re- 
quire processing. As shown in Figure 55, in this prior art 
the lane boundaries including lane boundary segments 
that have been lost sight of are actively determined 
based on lane boundary positions determined from an 
earlier image and the region of Hough space to be inves- 
tigated is determined based on the lane boundaries de- 
termined in the preceding frame. 

Japanese Laid-open Patent Application No. Sho 
62-155140 also teaches a method in which lane bound- 
ary positions are estimated from the processing results 
of the preceding cycle, while Japanese Laid-open Patent 
Application No. Hei 1 -1 61 403 proposes a method for us- 
ing vehicle movement information to estimate lane 
boundary positions in the currently input image from the 
positions according to the processing results in the pre- 
ceding cycle. 

The above and various other technologies so far 
proposed in connection with lane recognition and image 
processing are likely to err in lane recognition in actual 
traffic conditions when, for example, an obstacle such as 
a preceding vehicle comes into the camera field at a po- 
sition overlapping a region to be processed, as shown in 
Figure 56. 

A first object of this invention is therefore to over- 
come this problem. 

Further, Japanese Laid-open Patent Application No. 
Hei 4-36878 teaches a technique for detecting a vehicle 
lane (road). This prior art enables detection of the pres- 
ence of a lane to the outside of a broken line insofar as 
the broken line can be recognized. Since perfect recog- 
nition of a broken line as such is not possible under all 
driving conditions, however, uncertainty remains at the 
time of the lane pattern changes (see Figure 57). 

A second object of the invention is therefore to pro- 
vide a system for processing lane images for a vehicle 
capable of responding flexibiy to changes in the number 
of lanes even when the number thereof is unknown. 

According to a first aspect thereof, the present In- 
vention provides a system for processing lane images 
for a vehicle, including an image sensor mounted on the 
vehicle for viewing a roadway scene ahead of the vehicle 
to output image data of the roadway scene including a 
lane on which the vehicle travels, image data processing 
means for processing the image data to output lane in- 
formation, and lane recognition means for recognizing 
the lane on which the vehicle travels based on the output 



lane information. The characteristic feature of the system 
is that obstacle position determining means is provided 
for determining the position of an obstacle present in the 
roadway scene, and that processing region determining 

s means is provided for determining a region of the data 
image to be processed at least based on the detected 
obstacle position such that the region excludes the de- 
tected obstacle position. 

The present invention thus provides a system for 

io processing lane images for a vehicle which enables sta- 
ble and reliable vehicle self-navigation by setting the re- 
gions of the image to be processed such that the lane 
recognition is unaffected by the presence in the image 
of a preceding vehicle or other such obstacle and by se- 

is lecting the optimum lane in a road environment in which 
obstacles such as other vehicles are present. 

The foregoing and other objects and advantages of 
the invention will be more apparent from the following 
description of a preferred embodiment of the invention, 

20 which is given by way of example only and with ref erence 
to the accompanying drawings, in which: 

Figure 1 is an overall perspective view for explaining 
a system for processing lane images for a vehicle 
2$ according to the invention- 
Figure 2 is a block diagram showing particulars 
regarding the sensors of Figure 1 and the process- 
ing of the sensor outputs; 

30 

Figure 3 is a block diagram sjmilar to that of Figure 
2 for explaining the configuration of Figure 2 more 
in terms of function; •" 

35 Figure 4 is a block diagram showing the image 
processing hardware of the block diagram of Figure 
2 (particulars of the system for processing lane 
images for a vehicle); : 

40 Figure 5 is a main flowchart showing the operations 
of the system for processing lane images for a vehi- 
cle according to the invention; 

Figure 6 is an diagram for explaining the step in the 
*s main routine of Figure 5 for reading estimated vehi- 
cle movement data; 

Figure 7 is a flowchart of a subroutine of the main 
routine of Figure 5 for reading lane boundary posi- 
>o tions and storing the lane boundaries on the vehicle 
side as historical information; 

Figure 8 is a diagram for explaining the estimated 
lane boundaries in the "actual plane" ref rred to in 
~>s the subroutine of Figure 7; 

Figure 9 is a diagram for explaining the rule for deter- 
mining the vehicle position in the subroutine of Fig- 
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ure7; . 

Figure 1 0 is a diagram for explaining the determina- . 
tion of start points and end points in the step in the 
subroutine of Figure 7 for determining the start s 
points and end points of approximative straight lines 
from estimated lane boundary data in the form of 
multipoint lines; 

Figure 11 is a diagram for explaining the estimated to 
lane boundaries on the vehicle side and the selected 
start and end points; 

Figure 12 is a diagram for explaining the step in the 
subroutine of Figure 7 for using estimated locus data is 
at the time of most recent image input and the esti- 
mated locus data at the time of image input in the 
current cycle to correct the start points and end 
points to the positions at the time of image input in 
the current cycle; 20 

Figure 13 is diagram for explaining the step in the 
subroutine of Figure 7 for calculating the slopes and 
intercepts of the straight lines from the corrected 
start point and end point positions; 2s 

Figure 14 is a flowchart of a subroutine of the sub- 
routine of Figure 7 for calculating each (p, 0) in the 
image from the slope of each line in the actual plane 
. and storing the result as historical information; 30 

Figure 15 is a diagram for explaining the subroutine 
according to the flowchart of Figure 14; 

Figure 16 is a diagram for explaining camera param- 3S 
eters used in the subroutine according to the flow- . 
. chart of Figure 14; 

Figure 17 is a flowchart of a subroutine of the main 
routine of Figure 5 for reading distance to an obsta- 40 

cle; ...... 

Figure 18 is a diagram for explaining the step in the 
subroutine of Figure 17 for using estimated locus 
data at the time of obstacle detection and estimated 45 
locus data at the time of image input in the current 
cycle to correct obstacle positions to positions at the 
time of image input in the current cycle; 

Figure 19 is a diagram for explaining an image so 
inspection zone used in the step of the subroutine 
of Figure 1 7 for determining the presence of obsta- 
cles in the camera field; . V : 

Figure 20 is a diagram for explaining similar deter- 55 
mination in the camera field in the actual plane; 

Figure 21 is a diagram for explaining the step in the 



subroutine of Figure 17 for converting the positions 
of obstacles in the camera field from coordinates in 
the actual plane to coordinates in the image; 

Figure 22 is a flowchart of a subroutine of the main 
routine of Figure 5 for setting the processing regions 
in the current cycle based on historical information 
and obstacle positions; 

Figure 23 is a flowchart of a subroutine of the sub- 
routine of Figure 22 for setting the processing 
regions when no historical information is available; 

Figure 24 is a diagram for explaining why the middle 
portions of the processing regions set by the sub- 
routine of Figure 23 overlap; 

Figure 25 is a diagram for explaining the positions 
of the processing regions set by the subroutine of 
Figure23; 

Figure 26 is a diagram for explaining processing 
regions set by the subroutine of Figure 23; 

Figure 27 is a flowchart of a subroutine of the sub- 
routine of Figure 22 for setting processing forbidden 
regions when historical information is not available 
but obstacle data are available; 

Figure 28 is a diagram for explaining the subroutine 
according to the flowchart of Figure 27; 

Figure 29 is flowchart of a subroutine of the flowchart 
of Figure 22 for setting the processing regions when 
historical information is available but obstacle data 
are not available; 

Figure 30 is a diagram for explaining the subroutine 
according to Figure 29; 

; Figure 31 is a flowchart of a subroutine of the flow- 
chart of Figure 22 for finding the nearest obstacle in 
the vehicle's lane; 

Figure 32 is a flowchart of a subroutine of the flow- 
chart of Figure 22 for setting the processing regions 
when historical information and data on obstacles in 
the vehicle's lane are available; 

Figure 33 is a diagram for explaining the subroutine 
. according to Figure 32; . 

Figure 34 is a flowchart of a subroutine of the flow- 
chart of Figure 22 for setting processing forbidden 
regions when historical information and data on 
obstacles outside the vehicle's lane are available; 

Figure 35 is a diagram for explaining the subroutine 
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according to Figure 34; 

Figure 36 is a diagram for explaining the basis of the 
step in the main routine of Figure 5 for specifying 
Hough transformation constraints using, historical s 
information; 

Figure 37 is a flowchart of a subroutine of the main 
routine of Figure 5 for specifying Hough transforma- 
tion constraints using historical information; w 

Figure 38 is a diagram for explaining the subroutine 
according Figure 37; 

Figure 39 is a diagram outlining the subroutine is 
according Figure 37; 

Figure 40 is a graph showing the Hough transforma- 
tion results when the method according to the sub- 
routine of Figure 37 was applied to the road image 20 
of Figure 36; 

Figure 41 is a diagram for explaining the step in the 
main routine of Figure 5 for selecting representative 
lines from the Hough transformation results by clus- 25 
terihg; 

Figure 42 is a flowchart of a subroutine of the main 
routine of Figure 5 for selecting representative lines 
from the Hough transformation result by clustering; 30 

Figure 43 is a diagram outlining the subroutine 
according Figure 42; . 

Figure 44 is a diagram for explaining the basis of the 3S 
step in the main routine of Figure 5 for checking for 
misdetection of the boundaries of the vehicle's lane 
using road width; • 

Figure 45 is a diagram outlining the step in the main 40 
routine of Figure 5 for using road width to check for 
misdetection of the boundaries of the vehicle's lane; 

Figure 46 is a diagram for explaining the basis of the 
step in the main routine of Figure 5 for checking the 45 
matching between representative segments and an 
edge image; 

Figure 47 is a diagram for explaining ideal and actual 
curve recognition, assuming the basis of the step in so 
the main routine of Figure 5 for checking the match- 
ing between representative segments and an edge 
image; 

Figure 48 is a diagram outlining the step in the main ss 
/ routine of Figure 5 for checking the matching 
between representative segments and an edge 
image; '.. . 



Figure 49 is the first half of a flowchart of a subrou- 
tine explaining the step in the main routine of Figur 
5 for checking the matching between representative 
segments and an edge image, the explanation 
centering on comparison between candidate seg- 
ments and the edge image; 

Figure 50 is the second half of the flowchart of Figure 
49; 

Figure 51 is a diagram for explaining the subroutine 
according to Figure 49; 

Figure 52 is also a diagram for explaining the sub- 
routine according to Figure 49; 

Figure 53 is a diagram outlining the step in the main 
routine of Figure 5 for checking the matching 
between representative segments and an edge 
image; 

Figure 54 is a diagram for explaining the conversion 
of lane boundary candidate point positions from 
coordinates in the image to coordinates in the actual 
plane conducted as part of the step in the main rou- 
tine of Figure 5 for checking the matching between 
representative segments and an edge image; 

Figure 55 is a diagram for explaining image process- 
ing according to the prior art; : 

Figure 56 is a diagram for illustrating a defect of the 
prior art of Figure 55 and explaining a first object of 
the invention; and 

Figure 57 is a diagram for illustrating a defect of the 
prior art and explaining a second object of the inven- 
tion. 

Figure 1 shows an overall perspective view of the 
system for processing lane images for a vehicle accord- 
ing to the invention and loaded on a vehicle. As shown 
in this figure, the system is equipped with an image sen- 
sor such as a CGD (charge-coupled device) mono^ 
chrome TV camera 10. The CCDcamera (image sensor) 
10 is mounted above the driver's seat in the vicinity of 
the rearview mirror to capture a monocular view of the 
road ahead, the system is further equipped with a radar 
system 12 comprising of 2 millimeter-wave radars 
mounted at the front of the vehicle to detects the : pres^ 
ence of obstacles such as vehicles and other solid ob- 
jects from the waves they reflect. A yaw rate sensor 14 
is mounted near the center of the passenger compart- 
ment for detecting the angular acceleration of the vehicl 
around the gravitational axis (z-axis). In addition, a vehi- 
cle speed . sensor 1 6 constituted as a reed switch is 
mounted in the vicinity of the vehicle drive shaft (not 
shown) for detecting the vehicle travel speed, and a 
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steering angle sensor 18 is provided in the vicinity of the 
steering shaft 20 for detecting the steering angle. 

The steering shaft 20 is equipped with a steering an- 
gle control motor 22, the throttle valve (not shown) is 
equipped with a throttle actuator 24 constituted as a step- 
ping motor, and the brake system (not shown) is 
equipped with a brake pressure actuator 26 (not shown 
in Figure 1). With this configuration, the vehicle navi- 
gates automatically by controlling the steering angle in 
accordance with a calculated steering angle control 
amount, adjusting the vehicle speed by controlling the 
throttle valve opening, and, when necessary, applying 
the brakes. 

The aforesaid configuration is shown in detail in the 
block diagram of Figure 2. The output of the CCD camera 
10 is sent to image processing circuitry or hardware 30 
where straight line segments are extracted by edge de- 
tection and Hough transformation, and the result is for- 
warded through a bus 32 for storage in a common mem- 
ory 34. An image processing CPU 36 extracts lane 
boundary candidates from among the straight line seg- 
ments and stores them in the common memory 34. An 
image evaluation CPU 38 reads the stored values at pre- 
scribed times and uses them to determine the lane 
boundaries. The output of the radar system 1 2 is sent 
through a radar processing circuit 40 and the bus 32 to 
be stored in the common memory 34. A radar evaluation 
CPU 42 reads the stored values at prescribed times for 
detecting the positions of obstacles within a coordinate 
system. 

The outputs from the vehicle speed sensor 16 etc. 
are sent to a locus estimating CPU 44 for estimating the 
vehicle locus. An action plan decision making CPU 50 
produces a desired course based on the stored values. 
The desired course and the estimated vehicle locus are 
sent to a locus tracking control CPU 46 which decides a 
locus (desired course) tracking control amount. In addi- 
tion, the locus tracking control CPU 46 calculates the 
steering angle control amount and forwards it to a steer- 
ing angle control CPU 52 which drives the steering angle 
control motor 22 through a PWM (pulse width modula- 
tion) controller 54 and a driver 56. The amount by which 
the motor is driven is detected by an encoder 58 and 
used for feedback control. 

A speed/tracking control section of the action plan 
decision making CPU 50 calculates a desired vehicle 
body acceleration and forwards it to a vehicle speed con- 
trol CPU 60. The vehicle speed control CPU 60 drives 
the throttle actuator 24 through an accelerator stepping 
motor controller 62 and a driver 64 and drives the brake 
pressure actuator 26 th rough a brake solenoid controller 
66 and a driver 68. The amount by which the brake pres- 
sure actuator 26 is driven is detected by a pressure sen- 
sor 70 and used for secondary feedback control. 

Although omitted from the drawings in the interest 
of simplicity, the configuration also includes waveform 
shaping circuits and other sensor output processing cir^ 
cuits Figure 3 shows the functions of the block diagram 



of Figure 2. Figure 4 is a block diagram showing the im- 
age processing circuitry or hardware of the block dia- 
gram of Figure 2 (the particulars of the system for 
processing vehicle lane images according to the inven- 
5 tion). The image processing circuitry or hardware 30 of 
Figure 4 specifically includes image input digitizing hard- 
ware 30a, real-time edge detection hardware 30b, 
Hough transformation hardware 30c and an image data 
bus controller 30d. 

The operation of the system for processing lane im- 
ages for a vehicle will now be explained with reference 
to the flowchart of Figure 5. 

First, estimated vehicle movement data are read in 
S10. 

As shown in Figure 6, this is done by, for example, 
defining the system startup time as time 0 and estimating 
the movement of the vehicle in an absolute coordinate 
system whose origin is the position of the vehicle at this 
time. The vehicle's position (x, y) and its yaw angle 9 rel- 
ative to the axis of the coordinate system at time T are 
calculated by a locus estimating section (the locus esti- 
mating CPU 44 in Figure 2) using the speed and yaw 
rate detected by the vehicle sensors. Prior to image in- 
put, the image processing CPU 36 receives these data 
and converts them to information regarding the vehicle 
position and yaw angle at the time of image input. 

Next, in S1 2, a camera pitch angle correction value 
is calculated from the amount of vehicle pitch angle var- 
iation. 

This is for use in correcting the position and eleva- 
tion of the CCD camera 10, which change owing to 
change in the attitude (pitch angle) of the vehicle during 
acceleration or deceleration: Specifically, this is 
achieved by mounting height sensors at the four comers 
of the vehicle (these sensors are omitted from Figures 1 
and 2), determining the positional relationship between 
the vehicle and the road surface, and using the informa- 
tion obtained to correct the camera elevation, thereby 
preventing errors in mapping calculations between coor- 
dinates in the image and coordinates in the actual plane 
coordinates. 

In the following S1 4, the lane boundary positions es- 
timated by the image evaluation CPU 38 up to the pre- 
ceding processing cycle are read and the lane bounda- 
ries on the vehicle sides are stored as historical informa- 
tion. This information is an important factor in the setting 
of the processing region of the image input in the current 
cycle and the constraints on the straight lines detected 
by Hough transformation. 

This will be explained with reference to the flowchart 
of Figure 1. First, in S1 00 of this subroutine, the estimat- 
ed lane boundary data up to the preceding cycle are read 
from the image data estimation. In other words, the es- 
timated lane boundaries in the actual plane of an image 
data evaluation such as shown in Figure 8 are read. 

Next, in SI 02, a check is made as to whether lane 
boundary data are available. Since the result of this dis^ 
crimination is naturally NO at system startup, the prb- 
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gram advances to S104 in which a flag is reset to 0 to 
indicate "No historical information". On the other hand, 
if the result of the discrimination in the S102 is YES, the 
program advances to S 106 in which lane boundary data 
for both sides of the vehicle are derived from ail evalu- s 
ated estimated lane boundary point sequence data. The 
estimated lane boundary data evaluated by image data 
evaluation and the vehicle position are assigned IDs 
from the left side in accordance with the rule set out in 
Figure 9. 10 

. Next, in S1 08, the start points and end points of ap- 
proximative straight lines are determined from the lane 
boundary point. sequence data. As shown in Figure 10, 
this is conducted by selecting the second point of each 
set of estimated lane boundary point sequence data as is 
the start point and the third point thereof as the end point 
Figure 11 shows the estimated lane boundary on each 
side of the vehicle and the selected start and end points. 

In the following S110, the estimated locus data at 
the time of most recent image input and the estimated 20 
locus data at the time of image input in the current cycle 
are used to correct the start points and end points to the 
positions at the time of image input in the current cycle. 
As shown in Figure 12, the image data evaluation is con- 
ducted by converting the four points Left-Start, Left-End, . 2s 
Right-Start and Right-End in the relative coordinate sys- 
tem (Xr1 , Yr1 ) whose origin is the position of the vehicle 
at the input time T1 of the image which is the most recent 
image processing result used in the evaluation to points 
in the relative coordinate system (Xr2, Yr2) at the input 30 
time T2 of the image to be processed in the current cycle. 

In the next S1i 2, the. slopes and intercepts of the 
straight lines are calculated from the corrected start point 
and end point positions. This operation is explained in 
Figure 13. In the following S1 14, each (rho, 0) in the im- 35 
age is calculated from the slope and intercept of each 
line in the actual plane and the result is stored as histor- 
ical information, this calculation is conducted by steps 
S200 to S204.of Figure 14. 

To be specific, the left and right straight lines 40 

Left (slope A_L, intercept B_L) 

Right (slope A_R, intercept B_R) 
in the actual plane coordinate systemhaving the vehicle 
position at the time of current cycle image input as its 
origin are converted to the lines in the image, the (rho, 45 
6) values in the Hough transformation coordinate sys- 
tem, namely 

Left' (rho^L, 9_L) * 

Right' (rhb_R, 6_R) 
are calculated, and the results are stored as "historical so 
information. " (As shown in Figure 1 5, in this embodiment 
the origin of the Hough transformation coordinate system 
in the image is defined as the center of the lower edge 
of the image, the length of a line drawn perpendicular to 
line L from the origin is defined as rho, and the angle ss 
thereof measured counterclockwise from the lower edge 
of the image is defined as 9.) 

Camera parameters such as shown in Figure 16 are 



used in converting positions in the actual plane to posi- 
tions in the image. In this embodiment, camera height H 
= 1 1 50 (mm), camera elevation Th = 0.043 (rad) and lens 
focal distance F = 12 (mm). In addition, the imaging el- 
ement is a 1/2-inch CCD (512 x 512 dot), and Xa is set 
to 0.0115 (mm/dot) and Ya to 0.0095 (mm/dot), where 
Xa and Ya are parameters indicating the length which a 
single dot in memory assumes in the image plane of the 
CCD camera 1 0. The program then advances to S116 in 
the subroutine of Figure 7, wherein a flag is set to 1 to 
indicate "Historical information available". 

Returning to main routine of Figure 5, the distances 
to obstacles are read in S16. The term "obstacle" here 
refers mainly to the preceding vehicle but also includes 
fixed objects: 

The subroutine for this is shown by the flowchart of 
Figure 1 7. First, in S300, the most recent forward obsta- 
cle data are read from the radar data evaluation (radar 
evaluation CPU 42 in Figure 2). Next, in S302, a flag is 
initially reset to 0 to indicate 'No obstacle data", where- 
after a check is made in S304 as to whether or not ob- 
stacle data are available and the number of obstacles is 
greater than zero. 

If the result in S304 is YES, the program moves to : 
S306, in which estimated locus data at the time of obsta- 
cle detection and estimated locus data at the time of im- 
age input in the current cycle are used to correct obstacle 
positions to positions at the time of image input in the 
current cycle. This operation is explained in Figure 18, 
As shown in this figure, this is conducted by converting 
the position in the relative coordinate system (Xrl , Yr1 ) 
whose origin is the position of the vehicle at the input 
time T1 of the image which is the most recent image 
processing result used in the evaluation to points in the 
relative coordinate system (Xr2, Yr2) at the input time T2 
of the image to be processed in the current cycle. 

Next, in S308, obstacles in the camera field are 
found and if any obstacle is present in the camera field, 
a flag is set to 1 to indicate "Obstacle data available". As 
shown in Figures: 19 and 20, the procedure for finding 
obstacles in the camera field involves using an equation 
for converting the four points P1, P2, P3 and P4 at the 
left and right extremities of the near and far edges of an 
inspection zone of the image shown in Figure 19 from 
the image plane to the actual plane and calculating the 
equations y = AL • x + BL and y = AR • x + BR of the 
straight field boundary lines L and R passing through the 
pairs of points on the left and right, as shown in Figure 20. 

In this embodiment, the inspection zone is set to 
have a length extending from 5 m to 80 m forward of the 
vehicle (xN = 5 (m), xF "= 80 (m)). The Obstacle positions 
(x, y) are substituted into the L and R equations and the 
obstacle is judged to be within the camera field if y is 
between the so-obtained yL and yR and also within the 
length of th inspection zone. 

Next, in S310, the corrected positions of obstacles 
in the camera field are converted from coordinates in the 
actual plane to coordinates in the image. This operation 
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is explained in Figure 21 . The camera parameters men- 
tioned earlier are used in this conversion. (If the result in 
S304 is NO, the program immediately returns to the main 
routine of Figure 5.) 

Next, in S18 of the main routine of Figure 5, the 
processing regions in the current cycle are set on the 
image based on the historical information and obstacle 
positions. 

The subroutine for this is shown by the flowchart of 
Figure 22. First, a check is made in S400 as to whether 
or not historical information is available. If the result in 
S4O0 is NO, predefined left and right processing regions 
are set in S402. This operation is conducted according 
to S500 and S502 of the flowchart of Figure 23. 

In the flowchart,- first, as shown in Figure 24, left and 
right processing regions that overlap in the middle are 
set in S500. The overlapping of the regions in the middle 
is for enabling obstacle detection even when the vehicle 
straddles a lane boundary. Next, in S502, the set values 
are stored as the processing regions (Default-L, R). As 
shown in Figure 25, each of the left and right processing 
regions is represented to have a start point (sx, sy) at the 
upper left and to extend lateral and vertical distances 
(1x, ly) from the start point. 

Figure 26 shows the processing regions set in the 
foregoing manner. In this embodiment, the nearest side 
of the recognition distance is set at 5 m and the farthest 
side at 80 m. The positions of the nearest and farthest 
sides. in the image is calculated using an equation for 
converting from actual plane positions to image posi- 
tions. In this embodiment, 16-dot overlaps are provided 
to the left and right of the image center line.. This is to. 
accommodate vehicie tilt relative to the lane boundary, 
during ordinary lane change while driving on an express- 
way. Further, for avoiding the effect, of edge detection 
processing and other such filtering, 8-dot margins are 
provided at the left and right edges of the image. 

Next, in S404, a check is made as to whether or not 
obstacle data are available. If the result is YES, the pro-, 
gram advances to S406 in which processing forbidden 
regions are set with respect to the left and right process- 
ing regions based on the obstacle position. The obstacle 
referred to here is the preceding vehicle or the like read 
in S16 of Figure 5. The subroutine for this processing is 
shown by the flowchart of Figure 27. The operation is 
expiained in Figure 28. 

In the subroutine of Figure 27, the obstacle positions 
are divided. into left, right and center zones (S600), the 
nearest obstacle in each zone, is select ed.(S602), and 
processing forbidden. regions of prescribed width Ware 
set in the respective zones (S604, S606, S608), Since 
no historical information is available, the processing for- - 
bidden regions are centered on the obstacles. . 

Again returning to Figur 22, if S400 finds that his- 
torical information is available, a check is made in S408 
as to whether or not obstacle data are available, and if 
the result is NO, the program advances to S410 in which 
left and right processing, regions are set using a prede- 
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fined detection distance and the historical information. 
The subroutine for this processing is shown by the flow- 
chart of Figure 29 and the operation is explained in Fig- 
ure 30. 

s In Figure 29 : the predefined detection distance and 
the slopes and intercepts represented by left and right 
straight lines are used, first to set a left side processing 
region according to the direction of the slope by calcu- 
lating the intersection with the detection distance (S700, 

10 S702, S704 and S706) and then to set a right side 
processing region by similar processing. (S708, S710, 
S712 and S714). Since no obstacle is present in this 
case, the set regions are substantially the same as those 
that would be set in S402. 

is in Figure 22, if S408 finds that obstacle data are 
available, S41 2 makes a search for the closest obstacle 
in the vehicle's lane. The subroutine for this operation is 
shown by the flowchart of Figure 31 . 

In this subroutine, a counter value is initialized 

20 (S800), the value is incremented by 1 (S802), the y-axis 
value yn of an obstacle in the camera field is confirmed 
to be less than Ymax (value on the y-axis of the nearest 
obstacle in the vehicle's lane; explained later) (S804), yn 
is substituted into the historical information linear equa- 

25 tions to obtain x-axis values Lx, Rx (S806), a check is 
made as to whether or not the x-axis value xn of the ob- 
stacle falls between Lx and Rx (S808) and, if it does, a 
flag indicating that an obstacle is present in the vehicle's 
lane is set (S8 10). If it does not, values xn and Lx are 

30 compared (S81 2), whether the obstacle is to the left or 
right of the vehicle's lane is detected from the result and 
a flag is set indicating "Obstacle present outside vehi- 
cle's lane" (S81 4, S81 6, S81 8). 

It is then confirmed that all obstacles in the camera 

35 field have been found and, if any was, information on the 
x- and y-axes positions of the nearest object in the vehi-. 
cle's lane is stored, whereafter, the points of intersection 
Lxm, and Rxm with the historical straighUines are stored 
(S820, S822 and S&2A). The nearest obstacle is found 

40 because distant ones do not hinder the detection. 

The program then advances to S41 4 in the subrou- 
tine of Figure 22, where it is determined from the afore- 
said flag whether or not an obstacle is present in the ve- 
. hide's lane. If the. result is YES, left and right processing 

45 regions are set in S41 6 using the distance to the nearest 
obstacle in the vehicle's lane and the historical inforrha- 
. .lion. The subroutine for this processing is shown by the 
flowchart of Figure 32 and the operation is explained in 
Figure 33. . • 

£Q According to the procedures mentioned from S900 :' 
to S904 of Figure 32, the left and right processing regions 
are set as shown in Figure 33 so as to avoid a region in 
r: the vicinity of the obstacle, namely, a region in which the 
V obstacle interferes with detection of the lane boundaries. 

55. Thus in this way a processing forbidden or prohibited . 
range is set between the left and right regions with the 
obstacle as a reference. 

. In Figure 22, if S41 4 finds that no obstacle is present 



SID- *EP^J>697641 A2_l_> 



7 



13 



EP 0 697 641 A2 



14 



in the vehicle's lane, left and right processing regions are 
set in S41 8 using the predefined detection distance and 
historical information, in the same manner as in S410. 

Next, in S420, a check is made as to whether or not 
an obstacle is present outside the vehicle's lane and, if 
the result is YES, the program advances to S422 in which 
processing forbidden regions are set with respect to the 
established right and left processing regions based on 
the positions of obstacles present outside the vehicle's 
lane. The details of this operation are shown in the sub- 
routine flowchart of Figure 34 and explained in Figure 
35. Specifically, a discrimination is made as to whether 
the obstacle outside the vehicle's lane is to the left or 
right (S1000), the nearest obstacle is selected (S1002), 
and processing forbidden regions ot a prescribed width 
W are set (S1 004, S1 006). 

The program then moves to S20 in the main routine 
of Figure 5, where Hough transformation constraints are 
specified using historical information. This is aimed at 
eliminating the effect of irregularly appearing shadows, 
road markers and the like by using the historical infor- 
mation in image data evaluation to conduct preferential 
segment detection of at least the estimated lane bound- 
aries of the vehicle's lane. It is also for detecting new lane 
boundaries as they become visible at the time of lane 
pattern changes or the like. 

The purpose of this processing will first be explained 
with reference to Figure 36. As illustrated, when an im- 
age of a road including a shadow or mark is processed 
by Hough transformation, the shadow or mark makes it 
difficult to extract the lane boundaries. At the bottom of 
Figure 36 is shown the Hough transformation (rho, 6) 
mapping plane obtained when the image is processed. 
It will be noted that the peak corresponding to the bottom 
line on the right side of the vehicle's lane is hard to dis- 
cern. This is because a large number of peaks of about 
the same magnitude are caused by the road markers. 

For overcoming this problem, it suffices to estimate 
the position of the lane boundaries in the current cycle 
from the results in the preceding cycle and obtain Hough 
transformation results only in the vicinity of the corre- 
sponding peaks. However, since tracking only the same 
lane boundaries as in the preceding cycle makes it im- 
possible to discover new lane boundaries, it becomes 
difficult to cope with lane pattern changes and lane 
branching. The current step is for overcoming this prob- 
lem! 

The subroutine for this operation is shown by the 
flowchart of Figure 37. First, in S1100, a check is made 
as to whether or not historical information is available 
and, if the result is YES, the program advances to S1102 
and S1104 in which sorting areas for the left and right 
processing regions are specified. These comprise of pre- 
defined fixed areas and variable areas of prescribed size 
centered on the straight line data of the left and right side 
historical information. This is illustrated in Figure 38. 

In this embodiment, the variable areas are set to a 
width of rho : 32 [dot], 0:10 [deg] centered on (rho, 9) 



of the historical information. The fixed areas are set in 
the range of (rho : 1 00 - 480 [dot], 9 : 98 - 1 1 6 [deg]) and 
(rho : 100 - 480 [dot], 6: 65 - 82 [deg]): If S1100 finds 
that no historical information is available, two areas of 
predefined range are set in the left and right processing 
regions in S1 106 and S1108. Specifically, fixed areas are 
set in the same manner as above, while (rho : 10 - 380 
[dot], 9 : 110 - 1 79 [deg]) and (rho : 10 - 380 [dot], 9:1- 
70 [deg]) in place ol the variable areas. 

The foregoing will be amplified with reference to Fig- 
ure 39. Sorting areas are set in the Hough transformation 
(rho : 9) plane of the current cycle to a prescribed width 
centered on the historical information (rho, 9). As illus- 
trated, since new lane boundaries coming into view can 
be expected to appear at the horizon, i.e. since it is pos- 
sible to discover new road edges at values of 0 jn the 
vicinity of 90 [deg], the sorting areas are permanently set 
at fixed widths. (As will be explained later, straight lines 
are extracted from the largest to fiftieth largest peaks in 
the Hough transformation -processed variable and fixed 
sorting areas.) 

Figure 40 shows the results of Hough transformation 
when the present method is applied to the road image of 
Figure 36. It will be noted that the peak corresponding to 
the broken line can be easily distinguished. Thus the es- 
timated lane boundaries are preferentially extracted from 
the Hough transformation results using the historical in- 
formation from the image data evaluation and, similarly, 
preferential extraction is also conducted in the vicinity of 
the horizon where the probability of detecting new lane 
boundaries is high As a result, it is possible to detect the 
lane in which the vehicle is traveling with high stability 
while also maintaining flexibility in response to changes 
in the lane pattern or in the number of lanes. - 

The program proceeds to S22 in the main routine of 
Figure 5 in which the image is input and its edges are 
detected. Specifically, the CCD camera 10 (image sen- 
sor) views a roadway scene in front of the vehicle and 
the analog NTSC video signal output by the CCD camera 
(monochrome TV camera) 1 0 is converted to a 512x512 
pixels, 8-bit quantized digital image. The digital image is 
imparted with a time delay of the required n umber of lines 
(three lines in this embodiment) and input for edge de- 
tection processing. In this embodiment, a sobel operator 
was used for the edge detection procession. The edge 
detection processing results were output as a 512 x 512 
edge image represented at an edge intensity of 8 bits 
(256 magnitude, levels). No particular preprocessing for 
correction of pixel intensity or noise reduction was con- 
ducted. This is because the straight line extraction capa- 
bility of Hough transformation is very robust to noise and, 
further, because influence from shadows and road mark- 
ers other than lane boundaries is adequately prevented 
by th provision of the constraints at the time the results 
are selected. : ." 

Next, in S24, the edge points in the processing re- 
gion are Hough transformed. Specifically, in each of the 
left and right processing regions the edge points of the 
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edge detection processed digital image data having 
magnitude exceeding the threshold value SLD of the 
processing region concerned are Hough transformed 
(according to the "Ohtsu Threshold Value Determination 
Method." described, for example, in Image Analysis 
Handbook , P503 (University of Tokyo Publishing Socie- 
ty). The edge points in the processing forbidden regions 
are not processed. 

Next, in S26, representative lines are selected from 
the Hough transformation results of the left and right re- 
gions by clustering This is for picking out the lane bound- 
aries, i.e. the main line segments, by statistically sorting 
the group of detected straight lines obtained by Hough 
transforming a road image such as shown in Figure 41. 

Specifically, when Hough transformation, which 
amounts to macroscopic straight line extraction process- 
ing is applied to an actual road image, the sharply de- 
fined long lane boundaries appear as large peaks, as 
shown at the lower part of Figure 41 , and a large number 
of slraio/il lines are derived and detected. Because of 
this it is not possible to obtain the desired road lane 
boundaries merely by selecting the peaks in order start- 
ing with the largest. It is therefore necessary to sort the 
group of many detected straight lines based on some 
characteristic so as to narrow them down to only the rep- 
resentative straight lines. However, no completely effec- 
tive method of sorting (clustering) the detected straight 
lines obtained by Hough transformation can be found 
among the prior art furthermost neighbor method, cluster 
radius designation or K-mean methods. 

When these three methods were tested and com- 
pared, the K-mean method was found to produce the 
most natural cluster division results. In addition, with the 
K-mean method the number of clusters can be increased 
or decreased merely by changing the value of K, while 
with the other methods it is hot clear how many clusters 
will result until the processing is completed. In view of 
the foregoing : the clustering method for extracting road 
lane boundary values was based on the K-mean method, 
but the number of clusters K thereof was controlled 
based on the standard deviation within the clusters of the 
K number of clusters and the capability to unite proxi- 
mate clusters was added. 

S1200 to S1226 of the flowchart of Figure 42 indi- 
cate the operation for clustering the straight line group 
detected by Hough transformation. Figure 43 is a dia-. 
gram outlining the method used. The procedure, which 
will not be explained in detail, comprises of evaluating 
the clustering results by the, K-mean method based on 
the standard deviations within the individual clusters and 
repeating the processing while increasing/decreasing K 
(number of clusters). 

It is therefore not necessary to know the number of 
janes in advance and, moreover,, response can be made 
flexibly to changes in the lane pattern. As the standard 
deviation used as the yardstick for cluster division and 
the distance threshold value for uniting proximate clus- 
ters, there can be values determined by experiment for 



an actual lane boundary. Because of this, it is possible 
to achieve sorting well matched to the purpose. 

The program next advances to S28 in the main rou- 
tine of Figure 5, in which a check is made for misdetec- 
5 tiori of the boundaries of the vehicle's lane using road 
(lane) width. 

This is for checking based on the road width whether 
or not any lines difficult to detect, such as broken lines 
and worn solid lines, were misdetected. Since, as shown 
10 in Figure 44, the peaks of broken lines resulting, from . 
Hough transformation are smaller than those of solid 
lines (less than 50% of those of solid lines), they are dif- 
ficult to distinguish from noise, making them susceptible 
to misdetection. Therefore, as shown in Figure 45, the 

15 straight line nearest the vehicle is found from among 
those selected by clustering, an opposing estimated 
straight line at a distance equal to the prescribed road 
(lane) width is calculated, and a straight line near thereto 
is found by searching the Hough transformation results 

20 down to the smaller peaks. 

In this method, if the straight line found has already 
been detected, this means that this lane boundary had 
a fair degree of magnitude in the Hough transformation 
mapping plane. In such case, since the detected straight 

25 line can be considered reliable, the road width in the ac- 
tual plane is calculated from the two straight lines L1 and 
L3 and this new road width is.used thereafter. The meth- 
od can therefore deal with changes in road width or some 
degree of difference between the actual road width and 

30 the initially set value.; : 

The program proceeds to S30 of the main routine of 
Figure 5 in which the matching between the. representa- 
tive segments and the edge image is cheeked. . 

If the straight lines detected by Hough transforma- 

35 Won are used as they are as the lane boundaries, line 
segments which do not match with the actual lane 
boundaries will be. included at branch lanes leading to 
exit ramps and the like. For avoiding this, earlier technol- 
ogies such as the University of Maryland's DARPA road 

.40 edge detection image processing method for ALV (Opti- 
cal Engineering, March 1986, Vol. 25, No. 3, pp. 409 - 
414) have required . sequential processing . of small 
processing regions. . 

This has been an obstacle to shortening processing 

45 time because, it requires Hough transformation, which 
accounts for a large proportion of the processing time 
even when high speed hardware is employed, to be con-, 
ducted sequentially a large number of times. As shown 
in Figure 46, moreover, when small regions are 

50 searched, a method capable of coping with broken lines 
has difficulty coping with curves, and vice versa. In ad- 
dition, susceptibility to influence from shadows and road . 
markers is high, making matters difficult without macro- 
scopic information. . . 

55 While road edges can be recognized by straight line 
segment approximation in the case of an ideally shaped 
curve as shown at the top in Figure 47, most curves of 
an actual expressway are gentle (radius of curvature of 
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400 m or more) so that the image becomes the same as 
one for a straight road, as shown at the bottom in Figure 
47. Since the prior art therefore recognizes curved road 
portions as straight ones, sudden changes occur at the 
time of entering and exiting curves and these become a s 
cause for steering control delay and instability. For overr 
coming this problem, it is necessary to sequentially in- 
spect the road edges and conduct processing as if trac- 
ing the curves. As was explained earlier, however, when 
sequential inspection type processing using small re- io 
gions is employed, it is difficult to cope with broken lines, 
shadows and the like. 

In light of this, Japanese Laid-open Patent Applica- 
tion Nos. Hei 3-1 39706 and Hei 4-36878 teach systems 
in which the effect of shadows and other such noise is is 
eliminated by setting small processing regions at posi- 
tions estimated from the processing results in the pre- 
ceding cycle and curves are coped with by edge inspec- 
tion and the use of processing regions divided into to far 
and near sections. However, this method can handle 20 
only a prescribed number of lanes and is unable to detect 
new lane boundaries and lane branches that come into 
view when changes occur in the lane pattern. 

This embodiment therefore uses the method shown 
in Figure 48. S1 300 to S1 334 of the flowchart of Figures 25 
49 and 50 indicate the operation for checking candidate 
segments against the edge image based on this method. 
Figures 51 and 52 are diagrams for explaining the sub- 
routine according to Figure 49. Thus., as shown in Figure 
53, the road edges are detected by a heuristic method 30 
using the processing results of the preceding cycle. As 
shown in Figure 54, the candidate lane boundary point 
sequences are converted from coordinates in the image 
to coordinates in the actual plane. 

As a result of the foregoing, the adverse effect of 35. 
shadows and road markers is reduced, thereby enabling 
highly precise lane boundary edge detection processing. ■- 
Specifically, since the. Hough transformation makes it : 
possible to obtain the macroscopic straight lines by 
processing the whole of a large region at one time, there 40 
is no. need to limit the number of lanes in advance, so 
that the aforesaid method is not affected by broken lines 
or by breaks at curves and is moreover able to faithfully 
reproduce curves. 

Since the aforesaid method requires only detection 45 
of the center of edge point distribution on the scanned 
lines and simple rules for shifting the scanning center, it 
does not involve any complex conditional branching. 
Since line segments are obtained by cutting the repre- 
sentative lines selected by clustering after Hough trans- so 
formation at their intersections, the ensuing processing 
can be carried out independently for each divided line 
segment, which makes it possible to achieve high 
processing speed by processing the segments in paral- 
lel. The method is less susceptible to horizontal lines re- 5S 
suiting from ^shadows and the like than is the method of 
investigating and tracking edge points sequentially and 
is able to cope with broken lines and breaks without ne d 



for complex inspection rules. 

The program proceeds to S32 in the main routine of 
Figure 5 in which the geometric characteristics of the 
candidate line segments are determined. This involves 
determining the geometric characteristics of each seg- 
ment of the lane boundary candidates detected in S30 
with respect to the other such segments and attaching 
the determined characteristics to the output data. These 
characteristics include, for example, the angles at which 
each segment connects with other segments. 

Next, broken line discrimination is conducted in S34. 
This is for increasing road configuration recognition per- 
formance by discriminating between broken and contin- 
uous lines. The method used is described in detail in the 
Applicant's Japanese Laid-open Patent Application No. 
Hei 3-158976 and will not be explained f urther here. 

Next, in S36, the detected lane boundary data is out- 
put to the image data evaluation section (the image eval- 
uation CPU 38 in Figure 2) and the program returns to 
the beginning of the main routine (to S10). 

As explained in the foregoing, in this embodiment 
image processing regions are set based on historical in- 
formation and obstacle position so that lane recognition 
is not affected by the preceding vehicle and other obsta- 
cles. As a result, lane recognition errors do not occur 
when driving behind another vehicle, for example. In ad^ 
dition, since Hough transformation constraints are spec- 
ified using historical information, changes in the number 
of lanes can be coped with even in a driving environment 
with an unknown number of lanes. 

While the foregoing embodiment uses a monocular 
visual sensor, it is alternatively possible to use a binoc- 
ular visual sensor, which can be used to measure dis- • 
tance, making it possible to omit the front radars. 



Claims 

1. A system for processing lane images for a vehicle, 
including: 

an image sensor (10) mounted on the vehicle 
for viewing a roadway scene ahead of the vehicle to 
output image data of the roadway scene including a 
lane on which the vehicle travels; 

image data , processing means (30) for 
processing the image data to output lane informa- 
tion; and 

lane recognition means (36) for recognizing 
the lane on which the vehicle travels based on the 
output lane information; 

characterized in that: 

obstacle position determining means 
(12,40,42) is provided for determining the position 
of an obstacle present in the roadway scene; and 

processing region determining means (30) is 
provided for determining a region of the data image 
to be processed at least based on the detected 
obstacle position such that the region excludes th 
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detected obstacle position. 

A system according to claim 1 , wherein said 
processing region determining means (30) deter- 
mines the region such that the region is made up of 
a left half and a right half that overlap partially with 
each other. 

A system according to claim 1 or 2, further including: 
lane boundary position storing means (34) for 
storing a lane boundary position estimated up to a 
preceding image data processing as historical infor- 
mation; and 

said processing region determining means 
(30) determines the region based on the detected 
obstacle position and the historical information. 



lines obtained from the Hough transformation; 

defining means for defining an average in each 
cluster as new centers; and 

uniting means for uniting proximate clusters to 
defect said straight line. 

A system according to any of preceding claims 1 to 
6, wherein said image sensor is a CCD camera (10). 



10 



15 



4. A system according to any of preceding claims 1 to 
3, wherein said image data processing means (30) 
includes: 20 

area determining means for determining an 
area in the image ; 

edge image extracting means (30b) for 
extracting edge images in the image data of the 
area; 25 

Hough transforming means (30c) for carrying 
out Hough transformation on the edge images; 

straight line detecting means for detecting 
from one from among 

straight lines obtained from the Hough trans- 30 
formation; ■__".'-.'*""" 

and 

said lane recognition means recognizes the 
lane on which the vehicle travels based on the . 
detected straight line. . 35 

5. A system according to claim 4, further including: 

lane boundary position storing means for stor- 
ing a lane boundary position estimated up to a pre- 
ceding image data processing as historical informa- 40 
tion; and 

said area determining means determines a 
first area based on the historical information and a 
second area based on a position where a new lane 
boundary can be expected to appear. 45 



6. A system according to claim 5, wherein said area 
determining means determines the second area 
based on a position corresponding to a horizon in 

the roadway scene. 50 

7. A system according to any of preceding claims 4 to 
6, wherein said straight line detecting means 
includes; 

setting means initially setting centers of clus- 55 
ters of the straight lines obtained from the Hough . 
. transformation; 

. clustering means for clustering the straight 
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lines from lane boundary point 
sequence data 



I 



S108 



Use estimated locus data at time 
of most recent image input and 
estimated locus data at time of 
image input in current cycle to 
correct start points and end 
points to positions at time of 
image input in current cycle 



5110 



Calculate slopes and intercepts of 



straight lines from corrected start 
point and end point positions 



S112 
V 



— \ . 

Calculate each \p, 0) in image SJ14 
from slope and intercept of each 1 
line in actual plane and store 
result as historical information 

Set flag to indicate ~| / 

'Historical information available" y 

C RETURN) 
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FIG. 8 



Attribute indicating whether 
X r .broken or solid line 



Road edge 




Road edge 1 , 1 



Yr 



i 



i 

L 



Road edge 2 



9 ° Vehicle position at time of 



preceding image input 



Estimated lane boundaries in actual plane 



FIG. 1 1 



Left_End(x.y) 

Left_Start(x.y) 
Yr— 



Xr 



Right.End(x.y) 



A 



Right_Start(x,y) 



Estimated lane boundary on each side of 
vehicle and selected Start and end points 
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FIG. 14 



( START ) 



Convert slope A and intercept B in actual 
plane to slope Ai and intercept Bi in image 

Y = A * X + B -» y = Ai X- x + Bi 



Convert straight line y = Ai * x + Bi 
in image to (p, 6) in Hough transformation 
coordinate system 



-S200 



-S202 



Store image straight line data (Ai, Bi) and WS204 
(p, e) as historical information 



( RETURN ) \ 
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FIG. 15 



Straight 
line Left 



Xr 



Yr 



Straight 
line Right 



Actual plane coordinate system 
having vehicle position at current 
cycle image input time as its origin 



J 



Left side 
straight line 




Right side 
straight line 



Estimated lane boundaries in image 
(Historical information) 
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FIG. 16 



Lens focal distance F [ram] 



Camera elevation 
Th [rod] 



1 




Image plane (CCD) 
Camera height H [mm] 



Road surface 



Xi[mm] 




Image 
memory 



i i 
Xp[dot] 



X<* = Xi/Xp [mm/dot] 
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FIG. 17 



( START ) 



S300 



Read most recent forward obstacle data 
from radar data evaluation 








^_^S302 




Reset flag to indicate 
"No obstacle data" 








5304 



-Obstacle data available - 
jind number of obstacles greater; 
than zero ? 



yes 



S306 



Use estimated locus data at time 
of obstacle detection and estimated 
locus data at time of image input 
in current cycle to correct obstacle 
positions to positions at time of 
image input in current cycle 

~~ I ^5308 



Find obstacles in camera field and 
if any obstacle is present in 
camera field, set flag to indicate 
"Obstacle data available " 



'S310 



Convert coordinate positions of 
obstacles in camera field from 
those in actual plane to those in 
image 



( RETURN } 
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FIG. 18 



Use estimated locus data at time of 
obstacle detection and estimated locus 
data at time of image input in current 
cycle to correct obstacle positions to 
positions at time of image input in 
current cycle 



Time T2 

Position and orientation of vehicle 
in absolute coordinate system 
determined by locus estimation 
tr(x2, y2, 02) 

Time Tl 

Position and orientation of vehicle 
in absolute coordinate system 
determined by locus estimation 
tr(x1, yl. 01) 




Obst_2(x,y) 



■ ■• ■- / • • •'• 
• •. / ; • 

•■.•:/•■••■ 

Absolute coordinate 
system in actual plane 
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FIG. 19 



Farthest edge of P2 
decision zone 



Nearest edge of 
decision zone 




Decision zone in image 



FIG. 20 



Obstacle position 



Camera field 

Decision 



XN 
Yr. 



e- 



VL (x.y) 



zone 



Q_. 

pr 



/P4' 



yR 



■ 1 ■ 

/o 



— . 

' P3' 



Decision in camera field in actual plane 

• : Obstacle in camera field 

O : Obstacle outside camera field 
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FIG. 21 



Xr 



O 

ObstJ(xr.yr) 



Yr-^ 



O 



Obst.2 



Positions of obstacles 
in image 




Obstacles in camera field 
in actual plane 



Conversion to positions 
in image 



FIG. 23 



( START ) 



Set left and right processing regions 
within predefined recognition distance 
range so as to overlap at image middle 






Store values set f 
processing regions i 
Def ault_L (sx, sy, 
Defaults (sx. sy, 


or left and right 
as 

ix, ly) 
ix, ly) 



-S500 



3502 



(return") 
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( START ) 



FIG. 2 2 



5400 




Find nearest obstacle 
in vehicle's lane 



^,S4T4 

^Obstacle" 
>resent in vehicle's] 
lane ? 



S416 



yes 



Set left and right proce- 
ssing regions using distance 
to nearest obstacle in 
vehicle's lane and 
historical information 



5402 



Set predetermined left and 
right processing regions 



S4I0 



Set left and right proce- 
ssing regions using prede- 
fined detection distance 
and historical information 



Set left and right 
processing regions 
using predefined 
detection distance 
and historical 
information 

r~ 




Set processing forbidden regions with 
respect to established right and left 
processing regions based on positions 

of obstacles present outside 
vehicle's lane 



5404 




Set processing forbidden 
regions with respect to 
earlier set left and right 
processing regions based 
on obstacle position 



( RETURN ) 
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FIG. 25 



(sx. sy) O 
ly 



ix 



FIG. 26 



Farthest side of 
recognition distance-- - 



Nearest side of 
recognition distance - 



Image center 



Left processing region 



Right processing region 
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FIG. 27 

( START ) 



Use value xp of Obst_i(xp, yp) to divide obstacles into 
left, right and center zones 





^S602 


Select obstacle in each zone with largest yp value 
(nearest obstacle) 




^S604 


Center zone C 

With respect to the nearest obstacle Obst_n(xp. yp), set in 
each of the left and right processing regions a processing 
forbidden region of prescribed width W whose lower edge is 
yp and upper edge is the farthest side of the recognition 
distance (limited to within processing region) 






Left zone L 

With respect to the nearest obstacle Obst_n(xp, yp), set in 
the left processing region a processing forbidden region of 
prescribed width W whose lower edge is yp and upper edge is 
the^f arthest side of the recognition distance (limited to 
within processing region) 


I ^f: fin ft 


Right zone R 

With respect to the rieare 
the right processing region 
prescribed width W whose 
the farthest side of the r< 
within processing region) 


st obstacle Obst_n(xp, yp), set in 
a processing forbidden region of 
lower edge is yp and upper edge is 
ecognition distance (limited to 



C RETURN ) 
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FIG. 2 8 



Zone 

^(Overlapped center portion) 



Left processing 
region 




Image 
center 



Processing 
forbidden region 



Farthest side of 
recognition distance 




Right processing 
region 



Obstacle (xp f yp) 



Left processing 
region 



In this embodiment, W is calculated 
(usmp equation referred to earlier) 
as width m image corresponding to 2m 
(vehicle width plus allowance) at 
obstacle distance (position of yp 
m actual plane). 



Farthest side of 
recognition distance 

Nearest side of 
recognition distance 






i 


y 


LLLA 



Left processing 
region 



Image 
center 



^ x 



'Right processing 
region 
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FIG. 29 



( START ) 

Set left ■ 7 

processing region 




Find xl of point of 
intersection PL(xl,-yf) 
xl = Yf/A_L - B_L/A_L 



S704 



L 



Find xl of point of - 
intersection PL(xl, yn) 
xl = Yn/A_L - B_L/A_L 



sx.L = Default_Lsx 
sy_L = Default_L.sy 
lx_L = xl-Default_L.sx + dx 
ly_L = DefaultJ_.ly 



S706 



Set right 
processing region 



S710 



1 




Find xr of point of 
intersection Pr(xr, Yf) 

xr = Yf/A_R - B_R/A_R 



S712 

r 



Find xr of point of 
intersection Pr(xr, Yn) 

xr - Yn/A R - B_R/A_R 



L 



S714 



sx_R =■ xr-dx 
sy_R = DefaultR^y 
lx_R = Def ault-R.lx-(xr-Def ault_R.sx) 
ly_R = Def aultJUy 



( RETURN ) 
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FIG. 30 



Farthest side of 
recognition distance 1 1 j- - - 

Nearest side of 
recognition distance 

Historical information y 
Left side straight tine 
(A'-L B_L) 




xl xr 



dx 



Historical information 
Right side straight tine 
(A.R, B_R) 



dx 



Farthest side of 
recognition distance ~ 



Nearest side of 
recognition distance 



'i-J-L 

I 1 1 

Pi! J ■JPr 

— ^ I? — 



y 



Left processing region Right processing region 
Pr ocessing regions when historical 
information is available but 
obstacle data are not 



PI 





7Pr 







Farthest side of v , 
recognition distance'-' 



Nearest side of X 
recognition distance 

Historical information 
Left side straight line 
(A.L. B_L) 



Processing regions 
when A_R<0 
In this embodiment, dx is 4 dots. 



Historical information 
Right side straight tine 



For avoiding the effect of edge detection processing and 
other such filtering. 8-dot margins are again provided at 
the left and right edges of the image. 
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( START 



FIG. 31 



n=0. Ym*=Q\ —S800 




Calculate Lx. Rx by substituting yn of obstacle 
Obst_n(xn, yn) in camera field into linear historical 
information equotions 
Left side straight line Left : 
y = A_L * x + B.L 
Right side straight line Right : 
y = A_R ♦ x + B_R 




S806 



S808 



S8I2 



Ymax = yn, X = xn 
Lxm = lx, Rxm = Rx 

Set flag to indicate 
"Obstacle present in 
vehicle's lane" 




Obstacle found to be 
outside vehicle's lane 
on left 



S814 



S816 



Obstacle found 
to be outside 
vehicle's lane 
on right . 



Set flag to indicate 
"Obstacle present outside 
vehicle's lane " 



'S8I8 



iKtotal number of obstacles, 



S820 




in fieldj?. 
no 

-Obstacle present in- 
-vehicle's lane ?- 

yes 



S822 



Image position (xn, yn) of 
obstacle in vehicle's lane 



no 

■5824 



Store Obst_M(X, Ymax) of nearest obstacle 
m vehicle s lane. Also store intersection 
points with historical straight line Lxm, Rxm 



( RETURN ) 



Historical information 
left side straight line 
(A_L. B. L) 




— x 



yn- 



Historical information 
right side straight line 
(A_R, B_R) 
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FIG. 32 



( START ) 



Define points of intersection 
between y-direction position 
Yobs of Obst_M(Xobs, Yobs) of 
nearest obstacle in vehicle's lone 
and left and right historical 
straight lines as Lxm. Rxm 



-SQOO 



Set left processing region 

sxJL=DefauIML.sx 
sy_L=DefauhJL.sy 
lx_L=Lxm - DefaultJLsx + dx 
lyJL=Defauk_L.ly 



SQ02 



S<104 



Set right processing region 

sx_R=Rxm -dx 
sy_R=Default_Rsy 

lx_R=Default_3R.lx -(Rxm-Default_R.sx) 
Iy_R=Default_R.1y 



( RETURN 
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FIG. 33 



Position in image of object 
in vehicle's lane 



Historical information 
left side straight line 
(A_L. ELL) 




Yobs -P 1 - 



Historical information 
right side straight line 
(A_R. ELR) 



Farthest side of 
recognition distance 



Nearest side of 
recognition distance 




Processing regions when historical information and 
data on obstacle in vehicles lane are available 



For avoiding the effect of edge detection processing 
and other such filtering, 8-dot margins are again 
provided at the left and right edges of the image. 
As earlier, dx is 4 dots. 
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FIG. 34 



( START *) 



Use value of xp of ObstJ (xp, yp) to divide 
obstacles into left and right zones. 
(This was done earlier in procedure for 
finding obstacles in vehicle's own lane.) 



SJOOO 



Select obstacle in each zone with largest 
yp value (nearest obstacle) 



S1002 



■S1004 



Left zone L 

With respect to the nearest obstacle Obst_n(xp, yp), 
set in the left processing region a processing 
forbidden region of prescribed width W whose lower edge 
is yp and upper edge is the farthest side of 
the recognition distance (limited to within 
processing region) 



S1006 



Right zone R 

With respect to the nearest obstacle Obst n(xp, yp), 
set in the right processing region a processing 
forbidden region of prescribed width W whose lower edge 
is yp and upper edge is the farthest side of 
the recognition distance (limited to within 
processing region) 
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FIG. 35 



Historicol information dx 
Right side straight li ne , , 
(A_R. B_R) . ifdx 



Obstacle outside 
vehicle's lane 



Historical information 
Left side straight line 
(A.L. ELL) 




Left processing 
region 



Right processing 
region 



Processing 
forbidden region 



Farthest side of 
recognition distance 



Obstacle (xp, yp) 




V 

Left processing 
region 



In this embodiment, W is calculated 
(using equation referred to earlier) 
as width in image corresponding to 2m 
(vehicle width plus allowance) at 
obstacle distance (position of yp 
in actual plane). 
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FIG. 37 




( START ) 



Historical 
information 
avoilable ? 




SHOO 



no 





fyes f SW2 


Specify sorting area (predefined fixed area) 
for left processing region (Specify variable 
area of prescribed size centered on linear i 
data (p_L. 0_L)of left side historical 
information) 






Specify sorting area (f 
for right processing re 
area of prescribed size 
data (p_R, 0_R) of right 
information) 


•redefined fixed area) 
gion (Specify variable 

centered on linear 

side historical 



S1106 



Set two areas of predefined 
size and position for left 
processing region 



S1108 



Set two areas of predefined 
size and position for right 
processing region 



( RETURN ) 
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FIG. 40 




Hough transformation mapping plane 



;\D: <EP_^0697&41A2_I_> 
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FIG. 41 



SB 









Original image 




Group of straight lines obtained 
by Hough transformation 




Peaks in Hough transformation mapping plane 
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( START ) 



F 1 6.42 



Initial number of clusters K=4 



I 



Select 4 peaks starting with largest 
as centers of initial clusters 




pn=0 



] — S1204 



Find center nearest pn-th 
peak and define it as ele- 
ment of that cluster 



S1206 



S1208- 
S1210- 



pn=pn+1 | 



yes 



-Peak size>min_P- > . 
.and pn<50.?^^ 



SI 



12J2 
VJ1 




In this embodiment, 
min_P is calculated 
for Hough transform 
result in each pro- 
cessing region as 
min_P=Max peak*THR 



Define average of elements 
in each cluster as new 
cluster center 



i 



THR was set at 0.47. 
the value obtained 
empirically by comparing 
magnitudes of solid 
line and broken line peaks 



Define peak nearest center of 
cluster having largest standard 
deviation as center of (K + 1)-th 
cluster, and set K = K + 1 . 



Check whether centers of all 
of K number of clusters 
coincide with center in preceding 
cycle 




-S1214 



S1216 



Check variance inside cluster for all clusters 
and calculate standard deviation for each 



S1218 



- All standard deviations- 

- smaller than Lim_M ?- 



S1220 



S1222 




Lim_M = 0.01 
in this embodiment 



S1226 



Define largest peak in each cluster 
as representative peak 



Unite proximate clusters 
I 



( RETURN ) 



^51224 



In this embodiment, clusters whose 
representatives are less than 
0.15 apart are united 



.If* <rEPii_0697641 A2J„> 
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FIG. 46 




Broken line 



Search should be able to 
jump over breaks 



Incompatible 



\Situation is made 
even more difficult 
by shadow or 
the like 




Lane branching 



At break, search should also 
be able to ascertain connection 
with other direction 



FIG. 47 




Ideal curve recognition 




Actual curve recognition 
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[Outline of present method procedures] 
Processing region 



Road edge 
point sequence 



FIG. 48 




Representative line segments obtained 
by using processin9 region frame to 
cut candidate straight lines detected 
by Hough transformation 









L3 







Find point of intersection between 
candidate straight lines and divide here 




-or each divided line 
— segment, measure degree 
of matching with edge 
image 




Method of measuring degree 
of matching with edge image 



Span edge image over aiven p inH distribution renter 

seJmen a t° ng COnd?d0te "™ oT^flS 
segment scanned line and define , 

the same as detected 
point 




Define line segments with 
greater than prescribed 
degree of matching as 
lane boundary candidates 



For detecting curve into the distance, 
find amount of offset from candidate 
line segment of edge point distribu- 
tion center in scan of> preceding cycle 
and shift center of scan in next cycle 
by corresponding amount 





Obtain one-dimensional data 
by,converting to length in 
actual plane, calculate ratio 
of length of portion whose 
edge points were detected to 
overall length of candidate 
line segment, and define 
result as dearee of matching 
of extracted point sequence 



Output 
Point 
sequence 




Offset amount 



If no edge point is detected, 
return scan center to candi- 
date line segment 



Convert detected points to coordinates 
in actual plane and output as arbitrary 
point sequence data according to edge 
point distribution 
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( START ) 



FIG. 49 



S1300 



--'Does e of Li satisfy"* 
-45 i[deg]<«<!35 [deg] 



S1302 




yes 

I Scon verticouTT 
SJ304^ Y 



Storting from end point side having 
larper^ coordinate (if equal, from 
end pomt side having smaller jc 
coordinate), sequentially obtain 
scan data over of fsettable width 
2W ± 1 centered on line segment Li 



'S1306 



Calculate offset of each set of scan 
data as (Sum of coefficients of 
positions where edge points at or 
above threshold value were detected) 
/(Number of flags nf indicating 
detection of edge points at or 
above threshold value) 
(Round down decimal fractions) 



S1308 



Define sum of y_ coordinate at 
center of scan data in current cycle 
and amount of offset calculated 
in current eyrie as y' and store 
coordinate (x, v7 as center of edge 
distribution 



I 



Define sum of -offset -from Une 
segment Li in current cycle and 
offset amount calculated in 
current cycle as offset of scan 
data center from Li in next cycle. 
If number of edge detection 
flags nf is zero, return offset 
from Li to zero 




^Scanning to other 
*end point completed 7- 



SJ312 




no 



S1314 



SJ31&, 



Scan horizontally j 



I 



Starting from end point side having 
larger j£ coordinate (if equal, from 
end point side having smaller x 
coordinate), sequentially obtain 
scan data over of fsettable* width 
2W + 1 centered on line segment Li 



-S1318 



Calculate offset of each set of scan 
data as (Sum of coefficients of 
positions where edge points at or 
above threshold value were detected) 
/(Number of flags nf indicating 
detection of edge points' at- or 
above threshold value) 
(Round down decimal fractions) 



^S1320 



Define. sum of x coordinate at 
center of scan data in current cycle 
arid amount of offset calculated 
in current cycle as x f arid store 
coordinate (x; y ) as center of edge 
distribution 



X 



Define sum of of f set from line 
segment Li in current cycle and 
offset amount calculated in 
current cycle as offset of scan 
data center from Li in next cycle. 
If number of edge detection 
flags nf is zero; return of f set 
from Li to zero 




scanning to other 
~end point completed V 



S1324 




no 



55 



ID: <EP_^_0697641A2_L>. 



EP 0 697 641 A2 



FIG. 50 



Number of detected edge points 
in each set of scon data 



(A) 



0 1 2 1 3 1 3 | 0 foil 1 l0|0l2h \ Z 



1 



S1325 



Remove isolated noise by applying 
median filter to one-dimensional 
data obtained by lining up for each 
set of scan data the numbers of 
times edge points at or above 
threshold value SLD were detected 



-- *J In this embodiment, 
the filter size is 5 

i 

After filtering 



2|2|1 |Ol fo|o[ 1 1 1 1 - [ -1 



SJ326 



Calculate (Number of sets of scan 
data in which edge points at or 
above threshold value SLD were 
detected)/(Total number of sets 
of scan data Ns) and define result 
as degree of matching Ei of line 
segment Li with edge image 



The number of edge 
points detected in 
a each scan is 



not an issue here. r-fy 
The value of _n is rTZZ 
either 1 or 0, . 1 /' . ' 



3 ______ 




Ns 




S1328 



£is the sum of number of detected 
edge points, which do not have to 
be continuous. (Because isolated 
noise is removed by the one- 
dimensional median filter) 



Specify line segment Li as 
une segment to be output 



$1330 



Jl 



S1332 



S1334 



Remove line segment 
Li from line segments 
to be output 



Considering edge point distribution center 
coordinates stored in each set of scan data 
as lane boundary candidate point positions, 
convert them from coordinates in image to 
coordinates in actual plane. (As output 
points to be converted, select only 
points equal or nearest to predefined 
table of Y coordinates in image. 
In this embodiment, selection is made 
so as to obtain a sequence of; points 
spaced about 5m in actual plane) 



In this embodiment, 
the empirically determined 
value 0.15 was used as min.E- 
This is a level enabling 
detection of even quite badly 
worn broken lines 



coordinate 
able in image 



Q RETURN ) 



Become equally spaced 
in actual plane 
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FIG. 51 



(x1,y1) 



Candidate line segment Li(p, 0) 
Slope A, Intercept B 




y Processing region 

IV is 4 in this embodiment 



w 



w 



-4 


-3 


-2 




1 


2 


3 


4 



Position coefficients of scan data 
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FIG. 52 



Flags indicating detection of 
edge point of intensity at 
or above threshold value 



Ql I lololol 1 



In the diagram above, the offset from center is 

(-2 + 1 + 2 +3) / 4 = 1.0 

so that the next scan is offset 1 to the right. 



Candidate line segment 
y = A*x + A 




image 
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FIG. 55 



Detected lane boundaries 




Processing results in 
preceding cycle 

S \ 




Image input in preceding prjor flrt lnput in current cycle 



FIG. 56 



Wrong segment detected owing 
to effect of edge of 
preceding vehicle 




Prior art 



Processing results in 
preceding cycle 

/ \ 




Distance 
information 



Processing region 
according to invention 
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FIG. 57 




What was being recognized as 
a two-lane road became 
a three-lane road by addition 
of another lane on the right 
side 



Even though the lane pattern has 
changed, the new lane that has 
come into view cannot be detected, 
so that the vehicle cannot move 
to the lane on the right 
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(54) Lane image processing system for vehicle 

(57) A lane image processing system for a vehicle 
to recognize the lane on which the vehicle travels. The 
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image data of the roadway scene including the lane on 
which the vehicle travels. Obstacles such as preceding 
vehicle are detected and a region for image data 
processing is determined (S1 8) so as to avoid the posi- 
tion of the obstacle and the lane on which the vehicle 
travels is recognized based on the processed result. 
More specifically, lane boundary positions estimated up 
to a preceding cycle are stored as historical information 
(S14) and in the region Hough transformation is limited 
in areas determined based on the historical information 
and the horizon in the roadway scene. 
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FIG. 5 



Reod estwioted vehicle movement dotch ^— S/(7 



Calculate comer a pitct. angle correction value 
from amount of vehicle pitch angle variation. 



-512 



Read lane bounder/ positions estimated by image evaluation 
CPU 38 up to preceding processing cycle and store lane 
boundaries on vehicle sides os historical information 



Reoc distances to obstacles r ~-— - 516 



T 



SI4 



Set processing regions in current cycle bosed on 
historical information and obstacle positions 



-5/0 



Specify Hough transformation limits using 
historical information 



520 



Input imoge and detect edges} — -> 522 



524 



Corry out Hough tronsf or motion on edge points in processing region! 



Select representative lines from Hough trans- <^26 
by 



formation results of left and right regions 
clustering 



S28 



Check for. lone boundary misdetection using rood (lone) 
width . 



Check matcfcng between representative segments and edge 
mwge. (Calculate degree of matching with edge image and 
define point sequence on line segment with greater than 
prescribed degree of matching as lane boundary candidate 
point sequence. Lone boundary, candidate point 
sequences ore converted to coordinates in actual plane) 



Determine geometric characteristics of candidate 
line segments (Length, angle) 
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Discriminate broken lines 



S34 



Output detected tone boundary dotoH — 536 
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